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THIS RESEARCH PROJECT WAS UNDERTAKEN TO DETERMINE THE 
EFFECTS OF FOUR VARIABLES ON TEACHER SELECTION DECISIONS IN 
RELATION TO THE CONSISTENCY OF THE DECISIONS, THE FINENESS OF 
THE DISCRIMINATIONS MADE, THE TIME NEEDED TO MAKE THE 
DECISIONS, AND THE CONFIDENCE THAT THE DECISIONMAKER HAD IN 
HIS DECISIONS. THE FOUR VARIABLES WERE (1) AMOUNT OF 
INSTRUCTION PROVIDED IN HOW TO PROCESS INFORMATION, (2) 

NUMBER OF WRITTEN DOCUMENTS PRESENTED, (3) DEGREE OF MASKING 
OF INFORMATION, AND (4) INTERVIEW INFORMATION. THE EXPERIMENT 
WAS CONDUCTED IN A SIMULATED SITUATION USING 144 PRINCIPALS 
FROM THREE COUNTIES IN WASHINGTON AS SUBJECTS. EACH SUBJECT 
EXAMINED EIGHT FICTITIOUS APPLICANTS FOR A HYPOTHETICAL 
TEACHING POSITION AND MADE DECISIONS REGARDING THE 
APPROPRIATENESS OF EACH APPLICANT FOR THE POSITION. THE 
RESULTS OF THE STUDY SHOWED THAT--(l) INSTRUCTION REDUCED THE 
AMOUNT OF TIME IT TOOK TO MAKE DECISIONS, (2) THE SINGLE 
DOCUMENT REDUCED THE TIME IT TOjK TO MAKE DECISIONS AND 
INCREASED THE AMOUNT OF DISCRIMINATION IN MAKING ESTIMATES ON 
THE TEACHER EVALUATION INSTRUMENT, (3) THE DEGREE OF MASKING 
OF INFORMATION REDUCED THE ADDITIONAL INTERVIEW INFORMATION, 
INCREASED THE TIME NEEDED TO MAKE DECISIONS, INCREASED THE 
DISCRIMINATION ON ESTIMATING CONSEQUENCES ON THE TEACHER 
EVALUATION INSTRUMENT, AND INCREASED BOTH MEASURES OF 
CERTAINTY. THESE RESULTS AND THEIR IMPLICATIONS ARE DISCUSSED 
FOR THIS AND SUBSEQUENT STUDIES. THIS PAPER WAS PRESENTED TO 
THE ANNUAL MEETING OF THE AMERICAN EDUCATIONAL RESEARCH 
ASSOCIATION (CHICAGO, FEBRUARY 8-10, 1968). (HW) 




AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 
Annual Meeting, February, 1968 



SyTiposium : 

Information Variables Affecting Decisions Regarding the 
Selection of Teachers * 

Chairman; 

Glenn C. Boerrigter, U. S. Office of Education 



Participants : 

Situational Factors to be Considered in the Selection Process 
Dale Palmer, University of Washington 

Personality and Behavioral Characteristics Pertinent to Selection Teachers 
Michael Hickey, University of Washington 






The Format and Processing of Information in the Selection Process 
Donald Bauthues, University of Washington 



The Effect of Various ^nf ormation jor^^^ 
"Dale l 7 Bol'tohrUnlver^ 



Decisions to Select Teachers 









Discussant ; 

Julian C. Stanley, Johns Hopkins University 



* The research reported in this symposium is supported by the U. S. Office 
of Education, Grant No. OEC 4-7-061349-0266 

Making in the Selection sl Teachers. Director: Dale 1. Bolton. 



U.S OEPARlMEHl Of HEAllH. EOUCAllOH & WElfARE 
OffiCE Of EDUCiillON 



IHIS OOCUMEHl HAS lEEH REPROOUCEO EXACUY AS RECEIVED fROM IHE 
PERSON OR ORGAHiZAllOH ORIQIHAllHG II PO'HIS Of VIEW OR OPINIONS 
SIAIEO DO NOl NECESSARIIY REPRESENl OffICIAl OffiCE Of EOUCAIION 
POSIIION OR POLICY. 



THE EFFECT OF VARIOUS INFORMATION FORMATS ON DECISIONS TO SELECT TEACHERS 
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University of Washington 



This paper will present the purposes of the study, the variables involved, 
the experimental design, and the analysis procedures. It will 
some of the major results and implications of these results for practice and 

additional research. 



Purposes and Design of the Study 

Tile general purpose of the research project was to determine the effects 
of four variables, all related to information format, on teacher selection 
decisions in relation to the consistency of the decisions, the fineness 
discriminations made, the time needed to make the decisions, and the confidence 
o rerS "Lt th; decision maker had in his decisions. The 
variables manipulated were: (a) amount of instruction provided in how to pro 

cess Information, (b) number of written documents presented, (c) degree of 
making rinfor;ation, and (d) interview information. The variables involved 

in the study are shown in the following table: 





Table 1 
Variables Involved 


in the Study 


Independent Variables 


Dependent Variables 


Variable 
1. Instruction 


levels 

a. intruction 

b. no instruction 


Variable 

1. Time: total time taken 


2. Documents 


a. multiple 

b. single 


2. Discrimination: by grouping, and by 
estimated consequences 


3. Masking 


a. considerable 

b. partial 


3. Certainty: regarding ranking, and 

regarding estimated consequences 




c. none 




4. Interview 


a. audio-visual 
(filmed) 

b. audio (tape 
recorded) 


4. Consistency: regarding ranking, and 
regarding estimated consequences 




c. none 





The four dependent variables were selected because of their 
tangible gains for a school district: (a) time saved on a large 
teacher selection decisions will cumulate to considerable saving 



potential 
number of 
to a district; 
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(b) consistency in estimating outcomes of decisions helps to maximize long- 
range goals by reducing the discrepancy between estimates and actual outcomes 

(c) fine discriminations are needed for making subtle distinctions among 
teachers where the potential loss due to error is high; (d) uncertainty can 
lead to indecision and can cause vacillation and wasted motion. 



The independent variables were chosen because of their relationship to 
the way an administrator might use his time and how information might be pro- 
cessed! The use of a single summary document or partially-masked infomation 
(i.e., exceptional data rather than the total data available) might allow 
clerical help or data processing equipment to transform information into a 
more useful format and therefore permit the administrator to use time in 
actual decision making rather than cumulating and collating information. How 
ever, giving the decision maker instructions regarding how to process informa- 
tion might facilitate decisions to the extent that it would nullify the benefits^ 
of mechanical or clerical manipulation of information. If the interview informa- 
tion is beneficial, then it must be retained for the purpose of assisting in 
decision making. If, however, administrators do not benefit from interview 
information, perhaps the interview should still be retained--but for purposes 
other than decision making. 



The instructional variable was designed to give the subjects information- 
processing techniques. This was a two-level variable consisting of either 
instructions or no instructions. The instructions given were concise, and 
consisted of techniques which were assumed to be integral to effective and 
efficient information processing. These included: (a) withholding the deci- 

sion until all information had been considered, (b) scanning information to 
obtain a general impression, (c) clustering information items according to 
criteria established by the subject, (d) attending to exceptional information, 
and (e) consideration of the information in terms of the criteria established 
by the subject and the decision to be made. A short criterion test was taken 
by each subject who received instructions. 

Because the experiment was conducted in a simulated situation, some of 
the variables that might ordinarily affect administrative decisions could be 
controlled (e.g., assignment situation, supervision situation, evaluation pro- 
cedure, independence of decisions, physical conditions, time of year for the 
decision, order of presentation of subjects, order of presentation of informa- 
tion regarding applicants, and the moti.vation of the subjects.) 

The subjects used for this study were selected from three counties in the 
State of Washington. Districts were randomly chosen from these three counties 
and the first nine were asked to participate. These districts had slightly 
more than the number of principals needed for the study, and 1A4 were randomly 
assigned to the 36 treatments* The design was a completely randomized 2x2x3x3 
fixed model treatment arrangement, with measures on all four of the dependent 
varicMes . 



★The question of the validity of the decisions, or the "goodness" of the deci- 
sions in terms of whether the "correct" teacher was selected was omitted inten- 
tionally from this study. It was assumed that local school systems define 
teacher effectiveness according to specific local criteria; if so, the local 
system will be able to specify the outcomes desired in terms of teacher behavior. 




0 
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The general task performed by each subject was to examine eight ficti* 
tlous applicants for a hypothetical teaching position and make decisions re- 
garding the appropriateness of each applicant for the position. Each subject 
was asked to: (a) estimate how each applicant would be evaluated on a Teacher 

Evaluation Instrument at the end of a year of teaching, (b) rank order the 
eight applicants according to their desirability for the hypothetical situa- 
tion, (c) make an estimate of the certainty of his judgments regarding the 
prediction and the rank order by indicating how willing he would be to bet 
that his judgments were correct, and (d) group the eight applicants according 
to selected attributes or characteristics. 

The above tasks were completed during the morning session of the experi- 
ment. For purposes of measuring the consistency of the decisions, a retest 
was administered in the following manner: 

Five of the eight applicants presented in the first session were repeated 
in the second session. These five applicants were disguised by modifying 
certain minor data, e. g., age, birthplace, size, etc. Changes in makeup, 
lairpieces, and clothes altered appearances during the filmed interview. The 
other three ispplicants used during the first session were decoys and were re- 
placed by considerably different applicants during the afternoon session. The 
decoys appeared late in the order of presentation in the first session and 
early in the second session to aid in forming the impression that the second 
set was an entirely new set of applicants. It was assumed, that the insertion 
of the decoys did not affect the decisions -*igarding the five applicants on 
whom repeated measures were taken. 

After subjects had performed the same tasks with the second set of appli- 
cants, it was possible to obtain two measures of consistency: (a) a correla- 

tion between the first and second ranking of the five real applicants, and 
(b) a correlation between the first and second estimates of how each applicant 
would be evaluated on the Teacher Evaluation Instrument. 

Two measures of discrimination were computed after completion of the experi- 
ment: (a) the average number of groupings on the attributes selected (e.g., if 

one subject grouped the eight applicants into eight groups on a given attribute 
and another subject used three groups, then the first subject was considered 
to have discriminated more finely), and (b) the mean variance of the 16 appli- 
cant scores on each item of the Teacher Evaluation Instrument; the greater the 
variance, the more discriminating the individual; the smaller the variance, 
the less discriminating. 

An analysis of variance for the 2x2x3x3 factorial experiment was completed 
for the main and interaction effects of the four independent variables. By 
using the single measure of time, and two measures for each of the other depen- 
dent variables, there were seven ANOVAs computed. Tables for each of these 
analyses are provided in the Appendix. 
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Results of the Study 

The total results of the initial analyses are presented in Table 2. 

Since each analysis included the same independent variables, the sources of 
viriattof .r rte sL for .11 seven analyses. Table 2 indicates results that 
were significant at the .05, and the .01 levels. 

Table 2 

Results of Seven 2x2x3x3 Analyses of Variance for 
Four Dependent Variables* 



Dependent Variable s 



Diacrtmtnation Certainty Con,i,t^n ^ 



Source o f Variation 
1: Instruction 
2: Documents 
3: Masking 
4: Interview 
2 



Group- 

Jjig. 



Est . 

Cons. 



Rank- 
ing 



Est. 

Cons. 



Rank- Est. 
tng Cong,._ 



X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 



,01 

.05 

.01 

.01 

.05 



.05 

.05 

.01 

.05 



.01 



.05 



3 

4 

3 

4 
4 
2 
2 
3 
3 
2 



.05 



.05 



X 

X 

X 

X 

X 



3 

4 
4 
4 

3 X 4 



.05 



★Table entries are maximum probabilities 



Time 

■me analysis of variance, using time as a dependent variable, indicated 
the following results; 



^ No instruction required 18 minutes more time than instruction. 

2 : S^l^rie doiLnts required 12 minutes more time 

3. No masking required more time than partial masking, which eq 

more time than considerable masking. 

4. Audio-visual and audio interview Information were not significantly 
different,* but both took longer than no interview Information. A 
checrL^ linear trend indicated a significant linear «end at the 
.01 level of significance,** where the levels were in the order of 
noti6 % ftudio f And audio^visuHl • 



* A Newman-Keuls test was used for all post analyses of means. 

** Orthogonal comparisons were made for all tests for linear trend 
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5 The effect of Instruction interacted with the effect of documents 
in the following manner: When no instructions were given, the 

l^ltiple documents required 25.6 minutes longer; ho--"* 

instructions were given, there was “ t. 

quired. This interaction is plotted in Appendix B, Fig 

Dtacrimination 

Results of the analysis of variance, using the 8r°“Pi"8 
< ifl rhsracteristics as a measure of discrimination, yielded no d 
^cr ruS oflhe reSend^nt variables. (Two two-way interactions at the 

iitiigs on the Teacher Evaluation. Instrument as a measure of discriminat 
yielded the following results: 

1. Instruction had no effect on this measure of discrimination. 

9* Th(p ainele document produced more discrimination. 

So asking vLlded m«e discriminating result, than partial masking, 

5 ae effect ofinstruction interacted with the effect of 

the following manner: When no instructions were given, t**® 

ainriimpnta were less discriminating than the single documents. The 
tSwucUon appeared to depress the single documents' 

rrrtogrrt^prris^orthrLans indicated that P"® “-.tfu - 

were different were the multiple and single document, when no 

tions were given. This interaction is plotted in Appendix B, Exgure . 

Certainty 

fnr inAtructlon was not considered significant . ) , 

Audio-visual interview information yielded mors certainty than no 
fntervlertiformation. A linear trend existed in the of no 

lhe°!^1ecrof*tht°nimb^ interacted with the 

the interview information treatment in the folb /ing manner: e 

audio d^pre^sed the certainty scores for the mu ; P ® ^ 

treatment below P^?‘°^/2or; certainty with the audio-visual and no 
ment treatment exhibited more y <n Annendix B. 

interview information. This interaction is plotted in Appendix B, 

Figure 3. 



1 . 



2 . 



3. 
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The analysis of variance regarding the certainty of the ranking of appli- 
cants for the position yielded the following results: 

1. llie effect of instruction, documents, and masking was not significant 
on this measure of certainty. 

2. Audio-visual interview information yielded more certainty than 

audio information or no interview information. A linear trend existed 
in the order of no information, audio information, audio-visual 
information. 

3. The effects of the four variables interacted with regard to this 
measure of certainty. 



Consistency 

The analysis of variance, using as a measure of consistency the rank order 
correlations between event one and event two (morning and afternoon sessions) 
for the ranking of candidates for the position, yielded no differences for any 
of the independent variables. (One three-way interaction at the .10 level was 
not considered significant.) The variance within groups was so great on the 
consistency of rank order of the candidates from morning to afternoon session 
that differences did not appear. The small number of candidates (n * V®® . 

in the rank order correlation reduced the possibility of obtaining significant 
differences; likewise, this may partially account for the large variances in 
correlation coefficients within cells. However, similar within-cell variances 
were also noted for correlation coefficients between the two estimates of 
consequences on the A9 item Teacher Evaluation Instrument. The reason for 
such high-cell variance appears to be related to the heterogeneous backgrounds 
of the subjects involved, but further investigation of this phenomenon is 

warranted. 



The analysis of variance, using as a measure of consistency the corre- 
lation of the estimates of consequences on the Teacher Evaluation Instrument 
of event one with event two, yielded the following results: 

1. There were no main effects of the independent variables for this 
measure of consistency. 

2. The degree of masking interacted with the document variable in 

the following manner: The trend was for the single document to be 

more consistent as the amount of masking moved from considerable to 
partial to none, while the multiple documents tended to be more con- 
sistent as the amount of masking moved from none to partial to con- 
siderable. Orthogonal comparisons of means indicated that the single 
document yielded significantly less consistent results with con- 
siderable masking, significantly more consistent results with no 
masking, and was not different from the multiple documents with par- 
tial masking. The interaction is plotted in Appendix B, Figure 4. 
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Discussion of the Results 

Instruction, under these experimental conditions, reduced the amount of 
time it took to make decisions. Instruction also interacted with documents 
by reducing the time for multiple documents and by making the 
estimates mote discriminating on the Teacher Evaluation Instrument. Tnese 
results appear to warrant the recommendation* ® 

instructions be given to principals who are engaged in the selection of 

teachers. 

The single document reduced the time it took to make decisions and In- 
creasl^ thi Amount of discrimination in making estimates on the Teacher ^.lua- 
tion Instrument. However, documents interacted with interview information 
bnejren^rthe certamiy of estimate on the Teacher Evaluation Instrument 
for the audio interview information obtained by the multiple document treatm nt. 
DoUentralso interacted with the masking Information as far as consistency 
was concerned in estimation on the Teacher Evaluation Instrument. When "nltiple 
dncu^en^rwere used the most consistency was obtained with considerable masking, 
when the single document was used, it yielded the most consistency with no mask 
Ing. aese results appear to be compatible with a general notion that too much 
information (or information in an unmanageable forfi:) is confusing and pre ip 
tatrtnconsirten? responses, while too little Information precipitates random 

behavior. 

The degree of masking of information had a main effect of * 

linear fashion the time needed to make decisions. However, as indicated i^ 
fU nrinr section masking reduced the consistency of single documents. The 
Segrre of rasping’ airo deceased in a linear fashion the discriminations made 
in* the estimates*of consequences on the Teacher Evaluation Instrument. The 
results of the effects of masking combined with the results of the e ec s o 
t^e ioLmLL variable appear to warrant a reconunendation that no ^jasking be 
used in combination with a single document. This would "ot take ^^vant g f 
the saving of time found in masking of information, but it would ^ 

^e^on?v that is needed in estimation of consequences and take advantage of 
th^saving of time and increased discrimination caused by the single document. 

The linear trend for interview information indicated that additional 
interview information (moving from no information to audio information t 
l:rio-inu:i information) increased the time needed to ^ke 

creased the discrimination on estimation of consequences on the Teacher ^alua 

tion Instrument, and increased both measures of " the^Teacher 

these cases (where certainty was measured by ^ . . 

Evaluation Instrument) where a linear trend existed did a “^'‘..^ere 

the results of audio-visual and audio information. In this case, -nere 

ITlTso^ jrte«ction with the documents variable. Although there was a main 



*Reactions to the experiment solicited from the thtlr"'’*^ 

thought their responses in the morning session were different from 
feSs.I in the afternoon session. The results are being re-analyred with 

this idea in mind and conclusions may be different. 
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effect of the interview information on certainty of estimates on the 
Teacher Evaluation Instrument, the interaction of interview information with 
documents indicated that audio information depressed the certainty scores with 
multiple documents. The results of the study appear to warrant the following 
recommendations regarding interview information: 

1. Audio**vi8ual interview infonration should be used in spite of the 
increased time it takes. 

2. Administrators should not hesitate to use audio interview information 
where it is necessary or expedient (for example, telephone interviews 
with persons who are considerable distance from the location of 
employment). 

3. Additional investigations should be made of some of the other aspects 
of the interview information related to the audio treatment, e.g., 
the degree of the decision maker' s involvement in the interview, or 
conditions which allow maximuff concentration on the content of the 
interview may affect the decisions made. 

The results of this study were not concerned with the validity of the 
decisions. It was assumed that the importance of situational variables 
necessitates local validation of selection decisions and that this local valida- 
tion should be done with an optimum information format as far as time, certainty, 
discrimination, and consistency are concerned. Additional study in the simulated 
situation should allow us to determine whether subjects will be able to make 
valid decisions for predetermined and specified criteria with the information 
format recommended. 



Implications 

The implications (of the analysis made thus far) of this research project 
for practice in the selection of teachers are as follows: if principals or 

personnel directors involved in selection of teachers are of a similar nature 
to the subjects used in this study, their decisions regarding the selection of 
teachers will be affected by the format of the information. Further, one would 
expect that the single format that would yield optimum results as far as time, 
discrimination, consistency, and certainty are concerned would consist of: 
instructions regarding the processing of information, a single summary document, 
no masking of information, and interviews that Include visual as well as audio 
stimuli. 

The implications for further research that emerge directly from the 
results include the following: 

1. There is a need to know more about the strength of the instructional 
variable with other populations and with varying amounts and types 
of instruction. The main effect of this variabe as well as the 
interaction with the documents variable— is gratifying, considering 
that the total amount of instruction given to the subjects was less 
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than ten minutes. Will the same effect occur regardless of the 
experiential background of the subjects? Will it be necessary to 
give instructions each time the task is performed? Will the 
instructional effects transfer to other information-processing 
tasks? All of these questions warrant further study. 



2 . 



There is a need to know why the interaction between degree of mask- 
ing and single documents resulted in reduced consistency. Also, there 
is a need to know why the masking reduces the discrimination. Are 
these results due to the fact that not enough information is available 
to make discriminating and consistent decisions, or is it due to the 
lack of familiarity of principals with decision processes that empha- 
size attending to exceptional data? Studies need to be completed that 
will determine whether additional familiarity with exceptional -informa- 
tion-decision-making procedures yields different results. If so, time 
might be saved by using some procedure of partial masking. 



3 . 



There is a need to know why the nature of the interview information 
affects differentially the certainty of decisions made with different 
types of documents. If the certainty measured with the no interview 
information, multiple documents" was low in the same manner as the 
"audio information, multiple documents," a rather simple explanation 
is available. However, the reduction of certainty with the audio 
information and the lack of reduction of certainty with the no informa- 
tion is puzzling and needs further investigation. 



4 . 



Although the trend of the main effect of the interview information 
was generally compatible with the amount of the stimuli, the instances 
where the audio information was not significantly different ^ ® 

audio-visual information would imply that additional study should be ^ 
made of a direct comparison of the effect of these two levels on eci 

sions made. Likewise, it would be helpful , 

specific techniques of observation and listening skills would interact 

with the audio and audio-visual interview information. 



The implications of this research project (but not of the specific resu t ) 
are considerable in relation to increasing the knowledge of decision-maki g 
processes. An optimum information format will allow experiments to be con- 
ducted in a simulated situation without fear that the results wUl be unduly 
affected by the manner of presenting information to subjects. This experiment, 
then was Lcessary in order to utilize the simulated situation for testing 
hypotheses regarding certain elements of decision theory. For 

wa^s to describe the decision-making behavior of f In^whEra ” 

a set of individuals), or if one desires to prescri^ a manner in which a 
Lcision mre^might behave more effectively, it is necessary to determine both 
(a) the manner in which he predicts consequences, or at least what consequences 
he predicts! and (b) the value system he uses in the final cho ce But how can 
the*^prediction of consequences be separated from the values attached to ‘hem. 
How does one know, by observing the choice of a particular alternative, whet^r 
t^ choice was made on the basis of a high prediction of consequence and a low 
value, or the reverse, or both a high prediction of consequence and a high 

value? 
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Interest in value systems has led students of decision making to 
devise descriptive and prescriptive decision-making models. One intent 
of these models is to assist people in making the consequences they pre- 
dict and the values attached to them explicit, yet little work has been 
done to accomplish this intent. 

One approach to the description of the decision-making process might 
be to place subjects in a precisely described choice situation in which 
the consequences can be accurately determined and known by the subject. 

For example, a betting situation in which the odds were known— as in coin- 
flipping, rolling dice, or choosing combinations from a deck of cards-- 
might be used. Subjects could be taught the probabilities of certain con- 
sequences occurring, and their values could be inferred from the alterna- 
tives chosen, i.e., the types of bets they made. However, such an approach 
would leave much to be desired, because prediction of consequences would 
have been controlled, in a sense; therefore, one could only infer that 
differences in behavior were due to differences in value systems rather 
than the way consequences were predicted. Inability to determine concom- 
mitantly the subject's manner of predicting consequences and his value 
system is a limitation in this situation; such a limitation might elicit 
behavior considerably different from behavior in a less restricted decision 

situation. 

An approach from which broader generalizations might be made is one in 
which the situation is described, but the decision maker must make choices 
on the basis of his own prediction of consequences and attachment of values 
to these consequences. An exffinple is the simulated teacher selection situa- 
tion described in this set of papers. In this decision situation, subjects 
are not taught probabilities of consequences of choosing certain teacher 
applicants but must make estimates of what will occur if each teacher is 
hired. In addition, they must make choices among the teachers. The esti- 
mates of what will occur when a teacher is hired become the subject s 
explicit expression of probable consequences, and the value system of the 
subject is implied by this expression and his choices among teachers. 

The materials used to simulate the decision-making process for selecting 
teachers, then, provide a setting whereby descriptive and prescriptive 
theories of decision making may be tested. In addition, other variables 
that have been controlled in this experiment — especially those dealing with 
situational factors and interview inf ormation--can be manipulated in future 
experiments to determine their contribution to decision making. The results 
of this project, in addition to providing some recommendations for the 
practices of selecting teachers, make possible the control of a very impor- 
tant variable (viz. , the format of information) in future studies of the 
decision-making process. 
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APPENDIX Ao SEVEN ANALYSES OF VARIANCE USING INSTRUCTION, DOCUMENTS, 

MASKING AND INTERVIEW INFORMATION AS INDEPENDENT VARIABLES 

Al. An Analysis of Variance Showing the Effect on Time. 



Source of Variation 


d.f. 


Sums of Squares 


Mean Squares 


F 


1. Instruction 


1 


11826.6 


11826.6 


8.975 


2. Documents 


1 


5244.2 


5244.2 


3.98 


3. Masking 


2 


13828.5 


6914.3 


5.25 


4. Interview Informa- 
tion 


2 


116784.3 


58392.1 


44.3 


1x2 


1 


6601.6 


6601.6 


5.01 


1x3 


2 


531.1 


265.6 


m •• 


1x4 


2 


2273.4 


1136.7 


M m 


JL A *T 

2x3 


2 


2681.7 


1340.8 


1.02 


2x4 


2 


2387.1 


1193.5 


m » 


rfV "T 

3x4 


4 


5852.3 


1463.1 


1.11 


1x2x3 


2 


1724.6 


862.3 


m A 


1x2x4 


2 


487.5 


243.8 




1x3x4 


4 


775.0 


193.8 




2x3x4 


4 


1334.4 


333.6 




1 X 2 X 3 X 4 


4 


3239.5 


809.9 


“ * 


Within Cells (Error) 
Total 


108 

143 


142317.3 

317888.9 


1317.8 





* P i 

** p < .01 



A2. An Analysis of Variance Showing Effect on Discrimination, as Measured 
by Number of Groupings. 



Source of Variation | 


d.f. 


Sums of Squares ' 


Mean Squares ' 


F f 


t 

1. Instruction 


i i 


21.0 


21.0 


— 


2. Documents 


1 


0.2 


0.2 




3. Masking 


2 


1.8 


0.9 




4. Interview Informa- 


2 


40.0 


20.0 


tm » 


tion 

1x2 


1 


85.6 


85.6 


3.616 


1x3 


2 


37.5 


18.8 




1x4 


2 


143.0 


71.5 


3.022 


2x3 


i 2 


37.7 


18.8 


« •• 


2x4 


1 2 


12.1 


6.0 




3x2 


1 A 


95.4 


23.9 


« •• 


1x2x3 


j 2 


52.5 


26.3 




1x2x4 


2 


53.0 


26.5 




1x3x4 




55.0 


13.8 




2x3x4 


\ 4 


41.9 


10.5 




1 X 2 X 3 X 4 


n 

4 


108.7 


27.2 




Within Cells (Error) 


108 


2555.3 


23.7 




Total 


143 


3340.7 






y 
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A3« An Analysis of Variance Showing Effect on Discrimination, as Measured 
by Variance of Estimates of Consequences on the Teacher Evaluation 
Instrument 



' Source of Variation 


d.f. ' 


Sums of Squares 


Mean Squares 


F ' 


1. Instruction 


1 


0.09 


0.09 


m m 


2. Documents 


1 


0.77 


0.77 


4 . 06* 


3. Masking 


2 


1.55 


0.78 


4.09* 


4. Interview Informa- 


2 


1.94 


0.97 


5 . 10** 


tion 










1x2 


1 


1.02 


1.02 


5.37* 


1x3 


2 


0.25 


0.13 


m M 


1x4 


2 


0.38 


0.19 




2x3 


2 


0.05 


0.03 




2x4 


2 


0.71 


0.36 




3x4 


4 


0.98 


0.24 


M m 


1x2x3 


2 


0.13 


0.07 




1x2x4 


2 


0.17 


0.09 


«• m 


1x3x4 


4 


0.69 


0.17 


m «■ 


2x3x4 


4 


1.05 


0.26 


“ •• 


1 X 2 X 3 X 4 


4 


0.44 


0.11 




Within Cells (Error) 


108 


20.53 


0.19 




Total 


143 


30.77 







* P < 

*★ p ^ 



.05 

.01 



A4. An Analysis of Variance Showing Effect on Certainty of Estimates of 
Consequences on the Teacher Evaluation Instrument 



Source of Variation 



d.f. 



Sums of Squares 



Mean Squares 



1 . 

2 . 

3. 

4. 

1 

1 

1 

2 

2 

3 

1 

1 

1 

1 

1 



X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 



Instruction 

Documents 

Masking 

Interview Informa- 
tion 

2 

3 

4 

3 

4 
4 
2 
2 
3 
3 
2 



3 

4 
4 
4 

3x4 

Within Cells (Error) 
Total 



X J 

X 
X 
X 
X 



1 

1 

2 

2 

1 

2 

2 

2 

2 

4 

2 

2 

4 

4 

4 

108 

143 



10.56 

2.51 

2.26 

24.01 

0.34 
8.38 
13.54 
2.51 
23.35 
19.44 
3.18 
6.76 
16.08 
8.19 
5.28 
311.75 
458 . 16 



10.56 

2.51 

1.13 

12.01 

0.34 
4.19 
6.77 
1.26 
11.67 
4.86 
1.59 
3.38 
4.02 
2 Os 

1.32 

2.89 



3.66 



4.16* 



4.04* 



* p < 

** p < 




.05 

.01 



IS- 



AS. An Analysis of Variance Showing Effect on Certainty of Ranking of 



Applicants 



Source of Variation 


1 

d.f. 


Sums of Squares 


Mean Squares 


F 


1. Instruction 


1 


1.36 


1.36 


m m 


2. Documents 


1 


0.00 


0.00 


-- 


3. Masking 


2 


2.26 


1.13 


m « 


4. Interview Informa- 


2 


4 .06 


24.53 


6.02** 


tion 

1 X 2 


1 


2.78 


2.78 




1 X 3 


2 


5.01 


2.51 


m « 


1x4 


2 


14.39 


7.19 


tm ■» 


2x3 


2 


2.63 


1.31 


tm tm 


2x4 


2 


7.17 


3.5 


m « 


3x4 


4 


19.69 


4.92 


«■ tm 


1x2x3 


2 


3.93 


1.97 


m « 


1x2x4 


2 


5.72 


2.86 


-- 


1x3x4 


4 


31.86 


7.97 


-- 


2x3x4 


4 


18.33 


4.58 


-- 


1 X 2 X 3 X 4 


4 


52.44 


13.11 


3.218* 


Within Cells (Error) 
Total 


108 

143 ] 


440.00 

656.64 J 


4.07 


- 



* p < .05 

** P ^ .01 

A6. An Analysis of Variance Showing Effect on Consistency of Ranking of 
Applicants 



Source of Variation 


j 

d.f. 


SuL.s of Squares 


Mean Squares 


F 


1. Instruction 


1 


0.00 


0.00 


-- 


2 . Documents 


1 


0.02 


0.02 


mt a 


3. Masking 


2 


0.88 


0.44 


- - 


4. Interview Informa- 


2 


0.98 


0.49 




tion 










1x2 


1 


0.07 


0.07 


-- 


1 X 3 


2 


0.22 


0.11 




1x4 


2 


0.48 


0.24 


- - 


2x3 


2 


0.13 


0.07 


-- 


2x4 


2 


0.85 


0.42 


-- 


3x4 


4 


0.08 


0.02 


-- 


1x2x3 


2 


1.92 


0.96 


- - 


1x2x4 


2 


0.36 


0.18 


- - 


1x3x4 


4 


2.23 


0.56 


-- 


2x3x4 


4 


0.58 


0.14 


- - 


1 X 2 X 3 X 4 


4 


0.53 


0.13 


-- 


Within Cells (Error) 


108 


38.78 


0.36 


-- 


Total 


143 


48.10 








A7. 



An Analysis of Variance Showing Effect on Consistency of Estimating 
Consequences on The Teacher Evaluation Instrument 



Source of Variation 


d.f. 


Sums of Squares 


Mean Squares 


F 


1. Instruction 


1 


0.05 


0.05 


-- 


2. Documents 


1 


0.03 


0.03 


-- 


3. Masking 


2 


0.11 


0.05 


-- 


4. Interview Informa- 


2 


0.11 


0.05 


•• 1 


tion 










1 X 2 


1 


0.01 


0.01 


-- 


1x3 


2 


0.07 


0.04 


-- 


1x4 


2 


0.18 


0.09 


-- 


2x3 


2 


0.50 


0.25 


4s 18* 


2x4 


2 


0.02 


0.01 


-- 


3x4 


4 


0.45 


0.11 


c «■ 


1x2x3 


2 


0.18 


0.09 


-- 


1x2x4 


2 


0.11 


0.05 


-- 


1x3x4 


4 


0.23 


0.06 


-- 


2x3x4 


4 


0.36 


0.09 


-- 


1 X 2 X 3 X 4 


4 


0.07 


0.02 


-- 


Within Cells (Error) 


108 


6.41 


0.06 




Total 


143 


8.87 







APPENDIX B. 



Mean Time 



Figure 1. 



Mean 

Discrimination 



Figure 2 




The Interaction Effect of Instruction (I) and Documents (D) on 
Time. 




The Interaction Effect of Instruction (I) and Documents (D) on 
Discrimination , as measured by Variance of Estimates on the 
Teacher Evaluation Instrument. 



/ 
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Figure 3. 



The Interaction Effect of Interview Information (A) and Documents (D) 
on Certainty of Estimates on Tlie Teacher Evaluation Instrument. 




Figure The Interaction Effect of Masking (M) and Documents (D) on 

Consistency of Estimates on The Teacher Evaluation Instrument. 

er|c 



